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ABSTRACT 

Several  estimators  of  variance  of  the  ratio  estimator  in  simple  random 
sampling  have  been  proposed  in  Cochran  (1977),  Fuller  (1981),  Royall  and 
Cumberland  (1978),  Royall  and  Eberhardt  (1975),  Wu  (1982).  Their  performances 
are  compared  on  nine  populations  that  reflect  different  features  of  natural 
populations  encountered  in  practice.  One  criterion  is  the  mean  square  error 
of  the  variance  estimator  as  a  point  estimator  of  the  variance  of  ratio;  the 
other  is  the  reliability  of  the  associated  t-interval.  It  turns  out  that  the 
two  criteria  are  not  consistent.  The  apparent  contradiction  is  resolved  by  a 
conditioning  argument  on  an  ancillary  statistic,  i.e.,  the  reliability  of  the 
t-interval  can  be  predicted  by  the  closeness  of  the  corresponding  variance 
estimator  to  the  conditional  MSE  of  the  ratio  estimator  on  the  ancillary 
statistic.  Based  on  the  empirical  study,  the  jackknife  estimator  Vj  and  the 
estimator  v2  (and  other  asymptotically  equivalent  ones)  are  recommended. 

The  good  performance  of  these  estimators  is  attributed  to  their  ability  in 
"capturing"  the  ancillary  statistic. 
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SIGNIFICANCE  AND  EXPLANATION 


The  ratio  estimator  is  an  important  estimation  method  in  sample  surveys 
when  an  auxiliary  variable  is  available.  There  are  many  ways  of  estimating 
its  variance.  Their  empirical  performances  are  compared  on  nine  populations 
according  to  two  criteria.  The  more  interesting  one  is  the  reliability  of  the 
confidence  interval  based  on  the  t-statistic  (ratio  estimator-population 
mean) /estimated  standard  error.  The  jackknife  variance  estimator  and  other 
(asymptotically  equivalent)  estimators  are  recommended  for  practical 
purpose.  Their  good  performance  seems  to  be  related  to  inference  conditional 
on  an  appropriate  ancillary  statistic. 


The  responsibility  for  the  wording  and  views  expressed  in  this  descriptive 
summary  lies  with  MRC,  and  not  with  the  authors  of  this  report. 


ESTIMATION  OF  VARIANCE  OF  THE  RATIO  ESTIMATOR: 

AN  EMPIRICAL  STUDY 

Chien-Fu  Wu  and  Lih-Yuan  Deng* 

1 .  INTRODUCTION 

This  paper  concerns  the  estimation  of  variance  of  the 
ratio  estimator  under  simple  random  sampling.  While  the 
setting  is  simple,  we  hope  this  study  will  eventually  lead 
to  better  understanding  of  the  important  problem  of  variance 
estimation  in  complex  surveys.  In  fact  even  in  this  simple 
setting,  the  problem  of  choosing  "good"  variance  estimators 
is  unsettled.  More  than  a  dozen  estimators,  proposed  in  a 
span  of  some  thirty  years,  are  listed  in  Rao  (1969)  and 
Royall  and  Cumberland  (1981).  The  majority  of  estimators 
are  design-based,  i.e.,  their  justification  and  choice  are 
based  on  the  performance  according  to  the  probability 
mechanism  that  generates  the  sample.  A  few  others,  proposed 
by  Royall  and  his  collaborators,  are  model-based.  According 
to  this  approach,  the  inference  should  be  made  conditional 
on  the  observed  sample  and  a  hypothetical  superpopulation 
model.  The  sampling  design  becomes  irrelevant.  other 
estimators,  e.g.  the  jackknife,  may  not  be  justified 
exclusively  by  either  approach. 

Previous  work  on  the  comparison  of  variance  estimators 
for  ratio  include,  among  others,  Rao  and  Beegle  (1967),  Rao 
(1968,  1969),  Rao  and  Rao  (1971),  Rao  and  Kuzik  (1974), 

Royall  and  Eberhardt  (1975),  Royall  and  Cumberland  (1978, 

1981),  Krewski  and  Chakrabarty  (1981),  and  Wu  (1982).  The 
theoretical  comparison  of  various  variance  estimators  is 
made  by  assuming  that  the  x  and  y  populations  satisfy 
some  linear  regression  models  (superpopulations).  Although 
the  results  are  sometimes  exact,  dependence  on  the 
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superpopulation  parameters  can  be  delicate.  This  prompted 
Royall  and  Eberhardt  (1975)  to  study  the  model-robustness  of 
some  variance  estimators.  Their  definition  of  robustness  is 
restricted  to  the  bias  behavior  of  the  variance  estimators 
when  the  true  parameters  deviate  from  the  assumed  ones.  Wu 
(1982)  gave  the  first  model-free  comparison  of  some  variance 
estimators  by  expanding  the  estimators  and  working  on  the 
leading  terms  of  the  expansion.  Such  a  comparison  is  large 
sample  in  nature.  On  the  empirical  side,  comparison  is 
conducted  on  either  natural  populations  or  artificial 
populations  simulated  according  to  some  superpopulation 
models.  The  bias  and/or  mean  square  error  of  the  variance 
estimators  are  noted.  Motivated  by  the  prediction  theory 
approach,  Royall  and  Cumberland  (1981)  took  a  different 
approach  by  studying  the  conditional  behavior  of  the 
variance  estimators  as  a  function  of  the  x-sample  mean  x. 
They  showed  that  some  variance  estimators  can  behave 
drastically  different  over  a  range  of  the  x  values.  They 
further  argue  that  the  conditional  (on  x)  mean  of  a 
variance  estimator  should  closely  follow  the  conditional  (on 
x)  mean  square  error  of  the  ratio  estimator.  He  will  come 
back  to  this  point  in  §5  and  §6. 

The  work  to  be  presented  is  empirical  and  is  in  part 
inspired  by  the  very  stimulating  paper  of  Royall  and 
Cumberland  (1981).  Their  approach  to  empirical  study  can  be 
further  improved  in  three  respects.  They  did  not  consider 
estimator  v2  defined  in  §2,  which  is  motivated  by  the 
probability  sampling  theory  and  is  popular  in  practice.  In 
any  effort  to  criticize  the  more  traditional  sampling  theory 
approach,  it  seems  fair  to  consider  both  vQ  and  v2 .  See 
also  J.  N.  K.  Rao's  discussion  of  Royall  and  Cumberland 
(1978).  It  will  be  shown  later  that  v2  is  better  than 
Vq  in  several  desirable  respects.  Their  conclusion  in 
favor  of  the  prediction  theory  approach  could  have  been  more 
convincing  had  they  included  the  stronger  "rival"  v2  in 
their  study.  To  remedy  this  we  have  included  several 
additional  estimators  in  our  study.  The  six  natural 
populations  they  chose  look  artificial  in  that  they  are  all 
well  fitted  by  model  (2),  i.e.  straight  lines  through  the 


origin  with  increasing  residuals.  We  consider  nine 
populations,  six  identical  to  theirs  and  three  incorporating 
violations  of  three  key  assumptions  of  the  linear  regression 
model  (2)  that  typically  underlines  the  use  of  ratio 
estimator.  Detail  is  in  §4.  Besides  studying  the 
conditional  behavior  of  the  variance  estimators  in 
"tracking"  the  conditional  MSE  of  the  ratio  estimator,  an 
innovation  due  to  Royall  and  Cumberland,  we  also  study  the 
bias  and  MSE  of  the  variance  estimators  as  estimators  of  the 
unconditional  MSE  of  the  ratio  estimator  and,  more 
importantly,  the  actual  coverage  probabilities  of  the 
associated  interval  estimates  of  the  y-population  mean  Y 
as  compared  with  the  nominal  ones. 

2 •  RATIO  ESTIMATOR  AND  ITS  VARIANCE 

Suppose  that  a  population  consists  of  N  distinct 
units  with  values  (y^,x^),  where  >  0  for  1  <  i  <  N. 

A  simple  random  sample  of  size  n  is  taken  without 
replacement  from  the  population.  Denote  the  sample  and 
population  means  of  y^  and  xi  by  y,x  and  Y,X 
respectively.  The  ratio  estimator 


is  a  popular  estimator  of  Y.  It  is  simple  to  use  in 
practice.  It  combines  efficiently  the  covariate  information 
in  Xj^  when  y^^  and  x^  are  roughly  positively 
correlated.  It  is  the  best  linear  unbiased  predictor  of  Y 
under  the  following  superpopulation  model  (Brewer,  1963  ; 
Royall,  1970  ) 

yi  =  Sxi  +  Ei  '  (2) 


where  £ .  are  independent  with  mean  zero  and  variance 
2  1 

0  x^ .  The  ratio  estimator  possesses  other  desirable 
properties.  For  example,  it  is  robust  against  extreme 
values  in  the  individual  ratios  yVx^  (Rao,  1978). 
Traditionally  the  ratio  estimator  is  favored  over  the 
regression  estimator  mainly  for  computational  ease  in 
handling  large  data  sets.  Given  the  present  capacity  of 
computers  this  should  be  less  of  a  concern.  Fuller  (1977) 
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gave  examples  to  show  that  ratio  estimation  can  be  much  less 
efficient  than  regression  estimation.  We  believe  it  is  time 
that  more  attention  should  be  given  to  regression 
estimation. 

A  A 

There  is  no  closed  form  for  MSE  y  or  Var  y  .  Both 

R  R 

can  be  approximated  by  the  approximate  variance  (Cochran, 
1977,  p.  155) 


V 

appr 


1  -  f  _ J _ 

n  N  -  T 


(3) 


where  f  *  n/N  is  the  sampling  fraction.  For  large  samples 

the  approximation  is  adequate.  But  for  small  sample  size 

(n  <  12)  v_  _  can  seriously  underestimate  MSE  (Rao,  1968 
appr 

or  Cochran,  1977,  p.  164).  The  most  standard  estimator  of 
Vappr  ^ts  samP^e  analogue 


v 


0 


(4  ) 


Some  textbooks  mention  (but  not  endorse)  v2  as  an 
alternative  to  vQ , 


1  -  f  rX)2 

n  v  - ' 
x 


— 1 —  \  ry  -  i  x  y 

n  -  1  •:  '•yi  -  iJ 


-2 

The  original  motivation  for  v^  *  v2/X  as  a  variance 
estimator  of  the  ratio 


(5) 


R  -  Y/X 

is  the  unavailability  of  X.  Both  vQ  and  v2  are  easy  to 
compute . 

3 .  VARIANCE  ESTIMATORS  UNDER  STUDY 

Let  ei  *  yi  ~  Rxi  be  tb®  residual  from  the  straight 
line  connecting  (X,Y)  and  the  origin,  e^  ■  y^  -  rx, , 

r  »  y/x,  be  its  sample  analogue.  Apart  from  a  constant, 

2 

Vappr  tbe  P°Pu^at^on  roean  of  the  residual  square  e^. 

Estimation  of  vappr  can  be  vi®w®d  as  the  more  typical 

problem  of  estimating  the  population  mean  of  a  new 

2  - 

characteristic  e,.  By  taking  e.  a  e  .  ,  v„  can  be  viewed 

i  2  i  t  0 

as  the  sample  mean  of  e^  and  should  be  less  efficient  than 
the  ratio-type  estimators  v2  or 
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V1 

X  1  -  f  X  1  V 

-  0  n  -  n  -  1  t 

X  x  1 

-2 
e  . 
i 

(6  ) 

when 

xi 

2 

and  e^  are  positively 

correlated . 

A  general 

class 

of 

estimators 

V 

9 


(7  ) 


was  proposed  in  Hu  (1982).  He  proved  that  the  leading  term 

of  MSE ( v  )  is  minimized  by 
9 


gopt 


s2z 


x 

population  regression  coefficient  of 


zi  xi 

—  over  — 

Z  X 


(8) 


where 


z 


i 


N  N 

2  e  l  x  e  /  l  x  , 
1  1 


2 

S  and  S___  are  the  population  x-variance  and  ( x , z ) - 
x 

covariance  respectively.  The  second  term  of  zi  accounts 

for  the  possible  nonzero  intercept  in  the  population  when 

fitted  by  a  straight  line  according  to  (2).  A  (large- 

sample)  model-free  comparison  of  v2  and  vQ  readily 

obtains.  When  and  only  when  g  >  1,  v,  is  better 

op  t  ^ 

than  vQ .  It  may  be  easier  to  remember  and  to  interpret  the 
following  approximation  to  9opt  ( by  ignoring  the  second 
term  of  z^) 

g'  =  population  regression  coefficient  of 


N  E  .  e  . 

1  i 


over 


(9  ) 
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By  taking  a  sample  analogue  to  g 


g©pt  *  sample  regression  coefficient  of 


~  2  n  n 

*i  =  6i  "  2  «i  l  xiSi/  X  V 


i  n  ~ 

?  *i  ' 


we  obtain  an  asymptotically  optimal  estimator  v. 


within 


the  class  (7).  Similarly  we  can  take  a  sample  analogue  to 


g  =  sample  regression  coefficient  of 


- 1 n  „  2 
n  Zl6i 

and  obtain  another  estimator  v_. 

g 

Instead  of  making  a  ratio  adjustment  to  the  sample  mean 
„  2 

of  e^  as  in  v1 ,  Fuller  (1981)  suggested  a  regression 
adjustment  to  vQ  .  Denote  his  estimator  by 


V0  + 


b  (X  -  x) 
e  x 


where 


b  2  =  sample  regression  coefficient  of 

e  x 


e^  over  . 

By  standard  Taylor  expansion,  the  leading  term  of  v-  is 

v  and  their  asymptotic  behaviors  should  be  close, 

reg 

Another  estimator  of  interest  is  the  jackknife  variance 


estimator 


,,  -2  n  -  1  v  _2 

(1  .  f)  x  ___  l  D(j)  , 


where  D, 


is  the  difference  between  the  ratio 


(ny  -  y^)/(nx  -  x  ^  )  and  the  average  of  these  n  ratios. 
Royall  and  Cumberland  (1981)  and  Krewski  and  Chakrabarty 


(1981)  studied  the  model-based  and  sampling  properties  of 
v,.  Note  that  the  usual  justification  of  jackknife  is 
independent  of  a  superpopulation  model. 

Royall  and  Eberhardt  (1975)  suggested 


H 


x  X 
c 

'0  -2 
x 


(’  -?r 


■i 


(  14  ) 


when  x^  =  x-mean  of  non-sampled  units,  Cx  =  x-sample 
coefficient  of  variation.  Later  Royall  and  Cumberland 
(1978)  suggested  a  closely  related  estimator 


.  x  X 
f  c 


-2 

x 


i  11 

i  l 


(  15  ) 


1  - 


Both 


and 


vD  are  shown  to  be  unbiased  under  model  (2), 


approximately  unbiased  for  more  general  variance  patterns, 
and  asymptotically  equivalent  to  Vj. 

Another  variance  estimator,  which  follows  from  standard 
least  squares  theory,  is 


1 


,  x  X 
f  _c 

-2  n 
x 


„  2 
n  e 


-  1  \  X.  * 


It  is  unbiased  under  model  (2)  but  can  be  seriously  biased 
2 

if  var(y.  )  =  a  x^^  in  (2)  is  violated  (Royall  and 

Eberhardt,  1975).  Their  empirical  behavior  has  been  shown 
to  be  equally  bad  in  Royall  and  Cumberland  (1981).  For 
these  reasons  vL  will  not  be  considered  in  our  study. 

4 .  POPULATIONS  UNDER  STUDY 

The  preceding  variance  estimators  are  compared 
empirically  on  nine  populations  listed  in  Table  1.  The 
first  six  are  natural  populations.  The  original  data  were 
generously  provided  to  us  by  Professors  W.  G.  Cumberland  and 
R.  M.  Royall,  to  whom  we  wish  to  express  our  sincere 
thanks.  For  more  detailed  description  of  these  populations, 
see  their  1981  paper.  The  last  three  are  transformations  of 
population  1.  Their  description  follows.  The  first  six 
populations  are  plotted  in  Royall  and  Cumberland  (1981, 
p.  69-70).  Though  being  natural  populations,  they  are  all 


TABLE  1.  STUDY  POPULATIONS 


Population 

Description 

X 

y 

1 

Counties  in  NC,  SC,  and 

GA  with  1960  white  female 
population  <100,000 

Adult  white 
population. 

female 

1960 

Breast  cancer 
mortality,  1950-69 
(white  females) 

2 

U.S.  cities  with  1960 
population  between 

100,000  and  1,000,000 

Population, 

1960 

Population,  1970 

3 

Counties  in  NC,  SC,  and 

GA  with  fewer  than 

100,000  households  in 

1960 

Number  of 
households. 

1960 

Population,  excluding 
residents  of  group 
quarters,  1960 

4 

Counties  in  NC,  SC,  and 

GA  with  fewer  than 

100,000  households  in 

1960 

Number  of 
households. 

1960 

Population,  excluding 
residents  of  group 
quarters,  1970 

5 

National  sample  of  short- 
stay  hospitals  with  fewer 
than  1,000  beds 

Number  of  beds 

Number  of  patients 
discharged 

6 

Corporations  with  1974 
gross  sales  between  one- 
half  billion  and  fifty 
billion  dollars 

1974 

7 

Transformation  of  population 

1  (see  (17)) 

8 

Transformation  of  population 

1  (see  (18)) 

9 

Transformation  of  population 

1  (see  (19)) 

For  sources  of  populations  1  to  6,  see  Royall  and  Cumberland  (1981,  p.  68). 


-8- 


p 


well  described  by  straight  lines  through  the  origin  using 

weighted  least  squares.  The  squared  residuals  from  the 

fitted  line  increase  roughly  in  proportion  to  x.  More 

refined  models  like  a  +  0x  (linear  regression  with 

2 

intercept)  and  o  +  0x  +  rx  (quadratic  regression)  do  not 
differ  significantly  from  the  simpler  linear-through-the 
origin  model  Sx  except  possibly  for  population  5.  To 
represent  broader  range  of  real  populations,  we  construct 
populations  7,  8  and  9  from  population  1  to  reflect  the 
violation  of  three  key  assumptions  underlying  the  linear- 
through-the  origin  model  (2):  (i)  zero  intercept,  (ii) 

var(yi)  *  x^  (iii)  linearity  of  Ey^  in  x^  More 
precisely,  decompose  the  y^  value  in  population  1  ,  denoted 
old  y ^ ,  into 

old  y  =  Rx  +  (y^  -  Rx^) 

(  16  ) 


Define  the  new  y ^  value  in  population  7  as 

new  y.  =  old  y,  +  5  ;  (17) 

for  population  8, 

new  y±  =  yL  +  kxiei  '  (18) 

with  k  =  and  all  units  except  two  have  y^  >  0  ;  for 

population  9, 


new  y  . 

i 


6 ( x . -X ) 

e  1  ]  -  e.  , 


(19) 


where  8  =  S^1,  cQ  =  S^,  c1  =  0.1  +  exp[f)(  max  x^^  -  X )  ] 
so  that  y.  >  0  for  all  i.  1*i*N 

l 

Populations  1,  7,  8  and  9  are  shown  below. 

Some  characterisics  of  the  populations  are  given  in 

Table  2.  Note  that  x  and  y  are  highly  correlated 

(>  0.94)  for  populations  1-4,  6,  7.  The  x  and  y  of 

the  transformed  populations  8  and  9  are  less  correlated. 

Another  point  to  observe  is  that  can  be  smaller  or 

appr 

larger  than  MSE  for  sample  size  32.  There  is  no  systematic 
pattern  in  the  percent  underestimate  or  overestimate  (last 
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TAILS  2.  SOME  POPULATION  CHARACTERISTICS 


Population 

N 

PU,T>* 

100(Vappr  -  MSE) 

MSB 

1 

301 

0.947  • 

4.61 

4.71 

2.2 

2 

12S 

0.947 

113.9  a  10* 

111.9  a  10* 

-1.8 

3 

304 

0.998 

33.6  *  104 

32.4  >  104 

-3.6 

4 

304 

0.982 

230.9  *  104 

264.3  a  104 

14.6 

S 

393 

0.911 

1941.0 

1968.0 

1.4 

6 

331 

0.997 

30.7  *  1014 

33.7  a  1014 

16.3 

7 

301 

0.967 

77.7 

69.0 

-11.2 

8 

301 

0.803 

44.3 

47.6 

7.4 

9 

301 

0.824 

44.6 

43.0 

-3.6 

a.  Correlation  between  x  and  y  populations. 

b.  Baaed  on  1000  simulated  saaples. 
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column  of  Table  2)  in  V____  of  MSE  with  only  three  of  them 

appr 

over  ten  percent.  This  is  quite  different  from  Rao  (1968), 

where  he  found  that,  for  smaller  sample  sizes  n  *  4,  6,  8, 

12,  vappr  consistently  underestimates  MSE  with  average 

percent  underestimates  ranging  between  12%  and  17%.  The 

discrepancy  is  explained  in  part  by  the  difference  in  sample 

sizes.  More  importantly,  Rao '  s  computation  of  V,  and 

appr 

MSE  is  apparently  based  on  a  particular  superpopulation 
model  while  ours  is  model  free. 


5 .  RESULTS 

We  draw  1000  simple  random  samples  of  size  n  =  32 
from  each  population.  For  each  sample  we  calculate  the 

A 

ratio  estimate  yR  and  the  variance  estimates  vQ ,  v1 ,  v2 , 

vA  ,  v- ,  v  ,  v,,  v„,  v_  and  v  .  Note  v  is 

g  g  reg'  J  H  D  g  g 

”opt  opt  ’opt 

not  really  an  estimator  since  90pt  depends  on  the  whole 
population.  We  include  it  here  to  see  how  the  asymptotic 
results  in  Wu  (1982)  (or  §3)  predict  the  actual  performance 
for  sample  size  32.  The  MSE(y  )  in  Table  2  is  calculated 
1000  i  R 

as  1000  £  (y  Y)2  over  the  1000  simulated 

1  R 


samples.  For  each  variance  estimator  v, 

1000 


its  bias 


bias(v)  is  calculated  as  1000 


- 1 


1 


v  -  MSE(y  ) 

K 


over 


the  same  1000  samples,  and  its  root  mean-square  error 

.  1000  2  2  V 

/  MSE  (v)  as  (  1  000  £  (v  -  MSE  (  y  )  )  ) /2  over  the  same 

1  R 

1000  sample.  Results  are  given  in  Table  3. 

We  first  summarize  the  root  mean  square  error  behavior 
of  the  ten  estimators  in  Table  3  as  follows. 

(i)  The  asymptotically  optimal  estimator  v 


(pretending  g 


opt 


opt 

is  available)  is  the  best  or  nearly  the 


best  estimator  in  terms  of  minimizing  MSE,  as  well  predicted 
by  the  asymptotic  result  of  Wu  (1982). 


(ii) 


Among  v^ ,  v^,  v^ ,  the  best  performer  is 


consistently  the  one  closer  to  90pt*  For  examP3-e» 

g  *  1.59  in  population  1  is  closer  to  2  and  thus 
opt 

has  smaller 
v.  and  v„ 


r  MSE  (2.20)  than  those  (2.26  and  2.73)  of 
respectively.  This  is  again  predicted  by  the 


asymptotic  result  of  Wu  (1982,  §2.2). 
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Table  3.  Root  Man-square  error  and  bias  of  variance  estimators 


Bias  given  inside  the  parenthesis 


(iii)  The  estimators  v„,  v«  and  v  are 

g  g  reg 


asymptotically  equivalent  and  are  close  to 


They  all 


opt 


give  small  / MSE .  It  is  somewhat  surprising  that  vreg  does 
as  well  as  v 


and  better  than 


opt 


and  v-  on 

g 


populations  1-6  and  8.  Reasons  for  the  poor  performance 
of  vreg  on  populations  7  and  9  are  not  known.  The 


estimators 


are  more  stable  in  that  they 


v. ,  v-  and  v 
g  g  H 

perform  reasonably  well  for  all  the  populations, 
(iv)  The  jackknife  variance  estimator  v 7 


is  the 


worst  in  terms  of  MSE.  The  instability  of  v^  was  also 
reported  in  Rao  and  Rao  (1971),  Rao  and  Kuzik  (1974), 

Krewski  and  Chakrabarty  (1981).  The  performance  of  vD  is 
not  good  either. 

The  bias  of  each  variance  estimator  is  given  inside  the 
parenthesis  in  Table  3.  The  results  are  summarized  as 
follows . 


(i)  The  bias  is  usually  a  small  proportion  (say, 

<  30%)  of  the  total  /MSE  with  a  few  exceptions  for 

populations  3,  7,  8,  9. 

(ii)  The  estimators  vQ  ,  v^,  are  consistently 

downward  biased  for  estimating  the  MSE.  The  estimators 

v*,  v_  and  v  ,  being  close  to  one  of  v„ ,  v.  or  v„ , 
g  g  reg  *  0'  1  2' 

are  consistently  downward  biased.  Another  intriguing 
phenomenon:  among  vQ ,  v  ,  v2#  those  with  smaller  /MSE 

tend  to  have  bigger  (in  magnitude)  bias. 

(iii)  The  estimator  Vj  is  almost  always  upward 
biased,  while  vH  and  v^  exhibit  no  systematic  pattern. 

The  downward  biasedness  of  Vq  was  noted  in  Rao 
(1968),  Rao  and  Rao  (1971).  And  the  upward  biasedness  of 
Vj  was  noted  in  Rao  and  Rao  (1971).  Both  are  exact 
analytic  results  whose  validity  depends  on  some  particular 
superpopulation  models.  Model-free  (but  asymptotic)  results 
on  the  bias  of  v  ,  v  ,  v  ,  v  ,  v  have  been  obtained  by 
the  first  author.  They  will  appear  soon. 

In  estimating  the  population  mean  the  purpose  of 
variance  estimation  is  rather  for  assessing  the  variability 
of  the  ratio  estimator  than  for  estimating  the  variance 
itself.  A  more  interesting  and  relevant  criterion  is  the 
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behavior  of  the  associated  confidence  intervals  For  each 
variance  estimator  v  and  each  simulated  sample,  we 
consider  the  t-statistic 


and  the  (1  -  a)  confidence  interval  for  estimating  Y 

‘  V”1  ‘  V:11"  ^  12,1 

where  ta^2(31)  is  the  uPPer  ®/2  point  of  the 
t-distribution  with  d.f.  >31.  The  Monte-Carlo  coverage 
probability  of  the  confidence  interval  (21),  given  in  Table 
4,  is  calculated  as  the  percentage  of  the  1000  intervals 
(21)  that  cover  Y.  The  bias,  standard  deviation  and 
coefficient  of  skewness  of  the  associated  t-statistic,  given 
in  the  last  three  columns  of  Table  4,  are  based  on  the  1000 
t-values  (20). 

He  now  summarize  '('able  4  in  three  parts:  I.  normality 
of  t-statistic,  II.  width  of  t-interval.  III.  reliability 
of  t-interval  in  teswa  of  the  closeness  of  its  Monte  Carlo 
coverage  probability  a  the  nominal  one. 

I.  Except  far  populations  4  and  9,  the  bias  is  close 
to  zero,  the  s.d.  close  to  one  and  the  coefficient  of 
skewness  close  _o  zero.  Typically  the  t-statistic 
associated  with  the  estimator  Vq  is  not  normal,  especially 
with  its  large  coefficient  of  skewness. 

II.  Since  the  squared  length  of  the  t-interval  is 

proportional  to  the  expected  value  of  v,  from  E(v)  =  bias 

of  v  +  MSE,  we  can  use  the  bias  entry  of  Table  3  in 

assessing  the  width  of  t-interval.  Since  Vj  has  positive 

bias,  the  corresponding  t-interval  is  wider.  Similarly  vQ , 

v.,  v- ,  v. ,  v_ ,  v  all  have  negative  bias.  Their 

>  *  9  9  reg 

t-intervals  are  shorter.  The  intervals  associated  with 
vH  and  vD  are  in  between  the  two  extremes. 

III.  (i)  Generally  the  coverage  probability  is  lower 
than  the  nominal  level  1  -  a.  This  may  in  part  be 
explained  by  the  negative  bias  of  v  in  most  cases 
(except  Vj  and  some  cases  of  vH  and  vD)  The 
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0.78 


85.8  77.1  72.1  63.7  55.9 


97.3  92.2  84.3  73.4  65.9  -0.001  1.19  0.055 


97.2  88.9  79.7  65.5  S3. 2  -0.23  1.31  0.02 


95.7  90.5  85.5  76.8  70.3  0.23  1.23  0.96 


discrepancy  is  most  serious  in  population  4  where  the 
t-statistics  are  "abnormal",  and  is  serious  in  population  8 
for  1  -  a  =  0.7,  0.8,  0.9. 

(ii)  The  estimator  Vg  is  least  reliable  in  that  the 

actual  coverage  probability  is  far  off  the  nominal  level. 

The  most  reliable  one  among  the  nine  estimators  is  the 

jackknife  Vj  with  vD  as  the  second  best.  On  populations 

1  and  5-9  Vp  performs  as  well  or  nearly  as  well  as  Vj. 

The  performance  of  vH  is  comparable  to  vQ  (and 

sometimes  Vj)  on  populations  1,  2,  4,  5  and  7.  Other 

estimators  v  ,  v„ ,  v« ,  v.,  v  are  comparable  among  each 

1  2  g  g  reg  * 

other  but  trail  slightly  behind  vH. 

(iii)  Among  vQ ,  v1 ,  is  the  best,  v1  the 

middle  and  vfl  the  worst  for  most  cases.  A  partial 
explanation  is  that  v2  is  the  only  one  among  the  three 
that  is  asymptotically  equivalent  to  vJt  the  best 
performer . 

The  excellent  performance  of  Vj  might  be  explained  by 
the  large  expected  value  EvJf  or  equivalently  the  width  of 
the  associated  t-interval.  In  the  same  spirit,  could  the 
better  performance  of  v2  relative  to  v1  and  v1  to 
Vq  be  attributed  to  any  similar  behavior  in  their 
t-intervals?  As  remarked  in  II  they  are  all  short 
intervals,  but  from  Table  3  the  biases  of  vQ ,  v^,  v 2  do 
not  exhibit  any  clear-cut  pattern  to  support  such  claim.  We 
do  not  believe  that  the  length  of  the  interval  alone 
explains  the  difference. 

One  obvious  thing  to  observe  from  comparing  Tables  3 

and  4  is  that,  estimators  like  v„,  v_,  v  that  perform 

£  g  g  reg 

well  for  estimating  MSE(y  )  do  not  fare  very  well  for 
giving  reliable  interval  estimate.  On  the  other  hand, 

V,,  though  having  very  large  mean  square  error  for 
estimating  MSE(y  ),  is  extremely  good  in  giving  reliable 

K 

interval  estimate.  Perhaps  the  only  consistent  conclusion 
from  Table  3  and  4  is  that  Vq  fares  poorly  in  both 
criteria.  Since  a  variance  estimator  is  primarily  judged  by 
the  quality  of  the  associated  interval  estimator,  an 
important  question  thus  arises:  What  properties  of  v  as  a 
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point  estimator  will  provide  a  good  guide  in  judging  its 

performance  as  an  interval  estimator? 

We  propose  to  take  x  as  an  ancillary  statistic  and 

draw  inference  by  conditioning  on  x.  More  precisely  an 

estimate  of  the  conditional  mean  square  error  MSE ( y  | x ) 

R 

should  be  used  in  the  interval  estimate  of  Y.  In  the 
context  of  maximum  likelihood  estimation  Efron  and  Hinkley 
(1978)  proposed  a  version  of  conditional  variance  (given  an 
appropriate  ancillary  statistic)  for  constructing  reliable 
interval  estimate. 

To  see  how  different  variance  estimators  perform  in 
tracking  MSE (y_ I x) ,  we  divide  the  1000  samples  into  20 

K 

groups  of  50  samples  according  to  the  order  of  the  x 
values.  For  each  group  we  calculate  the  average  of  x. 


values.  For  each  group  we  calcul 
50 

£  x/50 ,  the  conditional  MSE  of 

1  50  i  2 

group,  J  (y  -  Y)  /50  and  the 

1  R  _  50 

nine  estimates,  v  =  V  v  /50  , 


within  the 


Y)  /50  and  the  averages  of  each  of  the 
50 

vn  =  I  v  /50  ,  etc.  We  then  plot  the 
1 


values  of  n/msE  ,  n/1^,  n/9^,  and  so  on,  against  the 

average  values  of  x  (the  factor  N  is  to  make  our  plots 
comparable  to  those  of  Royall  and  Cumberland  (1981).)  To 
save  space  we  only  show  the  plots  for  populations  2  and  3  in 
Figures  5  and  6.  Plots  for  other  populations  exhibit 
similar  patterns.  In  Figu; ■  s  5  and  6,  the  trajectories  of 


i v_ ,  f v  ,  v v  a 
g  reg  D 

the  trajectories  of 


v  are  omitted  because  they  are  too  close  to 
D 


respectively , 


One  can 


see  that  /v  ,  /v  and  /v  seem  to  track  the  conditional 

_ _  H  iJ  Z 

J MSE  (trajectory)  better  than  /v  ,  /v~  and  /^TT.  Such  a 

0  I  g 


rns>is  i traj ectory ;  Detter  tnan  rv  ,  kv  ana  /vA.  Such  a 

0  1  g 

visual  comparison  of  trajectories  is  somewhat  arbitrary  and 
imprecise.  Instead  we  consider  a  measure  of  distance  (22) 
between  the  /MSE-tra j ectory  and  any  /^-trajectory  given  by 


^20  |  ~  'MSE)2^  , 


where  the  summation  is  over  the  twenty  groups  of  values.  If 
the  distance  measure  for  a  variance  estimator  v  is 
smaller,  we  say  that  v  is  closer  to  the  conditional  MSE  of 
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O0) 


the  i 

ratio 

estimator. 

In 

nine 

populations  and 

nin 

From 

Table  5  we 

can 

nine 

estimators  as 

v-r'  vo  »  v  ~ 

J  2  g 

>  V« ,  V  >  V 

g  reg  1 

V 

< 

0 

ns  "better 

than"  . 

Again 

v0  is 

worst.  vH 

and  vD  are  s 

lightly 

and  v..  A 

g 

nd  v«  , 

g 

V 

r  eg 

are  the 

is  the  worst, 


VH'  VD  1 

where  M> "  mi 
v i  the  secom 
than  Vj,  v2 

performers.  This  is  in  general  agreement  with  the  results 
of  Table  4.  We  are  thus  led  to  the  tentative  conclusion 
that 

"Variance  estimators  that  estimate  the  conditional  MSE 
of  the  ratio  estimator  better  tend  to  give  more 
reliable  interval  estimates  of  the  population  mean." 


It  should  be  possible  to  justify  theoretically  this 
statement  at  least  by  assuming  a  reasonable  superpopulation 
model  between  y  and  x. 

The  apparent  contradiction  between  the  unconditional 
behavior  of  the  variance  estimators  as  point  estimators  of 
the  MSE  of  the  ratio  estimator  (Table  3)  and  the  reliability 
of  the  associated  interval  estimators  of  the  population  mean 
(Table  4)  is  now  happily  resolved.  If  the  inference  is  made 
conditional  on  an  appropriate  ancillary  statistic,  in  this 
case  x,  good  performance  of  an  estimator  for  estimating 
the  conditional  variance  often  points  to  good  performance  of 
the  corresponding  interval  estimator.  Therefore  our 
empirical  finding  lends  further  support  to  the  work  of  Efron 
and  Hinkley  (1978),  although  ours  is  for  finite  populations 
and  theirs  for  infinite  populations  and  parametric  models. 

6 .  CONCLUSIONS  AND  FURTHER  REMARKS 

Based  on  the  empirical  study  in  §4  and  §5  and  the 
theoretical  discussion  in  §3,  we  arrive  at  the  following 
conclusions . 

1)  The  estimator  vQ ,  (4),  is  the  poorest  among  the 

nine  estimators  considered  in  the  paper.  Its  t-intervals 
are  not  reliable  and  it  does  not  estimate  either  the  MSE  or 

a 

the  conditional  MSE  of  y  well.  However  it  is  the  most 

K 
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commonly  recommended  estimator  in  virtually  every  textbook 
on  sampling.  (In  fact  some  well-known  textbooks  do  not  even 
mention  the  better  estimator  v2 .  )  The  "de-mystification" 
of  Vq  is  probably  the  most  useful  of  all  the 
recommendations  made  in  our  paper. 

2)  Among  vQ  ,  v^ ,  v2 ,  v2  is  better  than  v^  and 

v1  better  than  vQ  for  giving  reliable  t-intervals.  The 
performance  of  vQ  ,  v1  and  v2  for  estimating  MSB  depends 
on  the  underlying  populations  and  has  no  direct  bearing  on 
the  performance  of  interval  estimates. 

3)  If  more  complicated  computations  are  allowed  (such 

i  may  be  an  issue  for  large  scale  surveys) f  we  have  more 

choices.  The  jackknife  Vj  gives  very  reliable  t-intervals 
and  vH>  vQ  are  almost  as  good.  Mote  that  for  large 
samples,  all  three  estimators  are  close  to  v2 ,  but  not  to 
any  other  vg,  g  *  2.  The  reason  that  Vj  does  so  poorly 
for  estimating  MSE  is  because  it  estimates  the  conditional 
MSE  well,  and  typically  the  conditional  MSE  varies  greatly 
with  x.  This  instability  of  Vj  for  estimating  the 
unconditional  MSE  has  also  been  reported  in  previous  papers 
but  should  not  concern  us  any  more. 

4)  The  estimators  v„ ,  v .  and  v  are 

i  g  g  reg 

asymptotically  equivalent.  They  are  good  for  estimating  the 
unconditional  MSE  but  are  mediocre  for  giving  reliable 
t-intervals. 

|  We  emphasize  that  reliable  t-intervals  seems  to  be 

|  related  to  the  good  performance  of  v  for  estimating  the 

conditional  MSE.  The  problem  of  choosing  a  proper  ancillary 
statistic  and  making  inference  condit  nal  on  it  is  an 
important  one  in  the  theory  of  survey  sampling. 

!  Encouraged  by  the  relatively  good  performance  of  v2 

over  Vq ,  we  have  considered  the  variance  estimation 
problem  in  other  settings.  For  the  regression  estimator 
under  simple  random  sampling, 

8 

ylrt  *  y  +  b( x  -  x),  b  «  — 

»* 

x 
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the  typical  estimator  of  VarCy^)  is  (Cochran,  1977  , 
p.  195) 

- - _  __  I  (y±  -  y  -  b<xi  -  x))  , 

which  is  the  sample  analogue  of  the  approximate  variance  of 
y^.  it  is  natural  to  consider  the  following  class  of 
estimators 

(— ) 9var ,  especially  g  =  2  . 
x 


A  detailed  report  will  be  available  later. 

In  stratified  random  sampling  with  small  sample  size 

per  stratum,  the  combined  ratio  estimator  is  often  used. 

Let  W.  *  N.  /N  be  the  hth  stratum  weight,  y.  ,  x.  ,  y.  ,  X 
n  n  n  n  n 

be  the  y-  and  x-sample  and  population  means  of  the  hth 
stratum.  The  combined  ratio  estimator  is 


Rc 


—  x, 

Est 


=  l  w 

It  “ 


h  h 


=  y  w.  x,  . 

it  “  h  h 


Its  approximate  variance  is 


L  1 

l  - 


1  nh 


4 

— 1 —  yh  ( 


;yhi  -  5h  - 1  ‘-hi  -  V’2 


The  following  class  of  estimators 

L  1_fK  i  ni 
y _ h  1  yr 

1  nh  "h'1  1 


-  l  i-r.  ,  n. 

c  X  'igv  h  1  vh, 

l  l  (yh.  - 


St 


St 


St 


( X  . 

hx 


x  )  ) 
n 


has  been  considered  by  the  first  author  (CFW).  The  case 
g  -  2  is  of  special  interest.  The  detailed  results  will 
be  reported  elsewhere. 

Extensions  in  other  situations  are  obvious. 
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